Apparatus, method, and storage medium

ABSTRACT

An apparatus includes an extract unit configured to extract features of a first image based on an electromagnetic wave in a first frequency band, an acquire unit configured to acquire motion information about the features, a classify unit configured to classify the features into a first group and a second group based on the motion information, and a remove unit configured to remove, from the first image, a signal corresponding to the feature belonging to the first group.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The aspect of the embodiments relates to an apparatus, a method, and astorage medium.

Description of the Related Art

Japanese Unexamined Patent Application Publication (Translation of PCTapplication) No. 2007-517275 discusses a method for causing each personto stop at a predetermined position, for example, at a gate forcontrolling each person to enter a building or leave from the building,or at an entrance of an escalator, irradiating the person with anelectromagnetic wave, and detecting an object owned by the person basedon the result of reception of the electromagnetic wave reflected by theperson.

SUMMARY OF THE DISCLOSURE

An apparatus according to an aspect of the embodiments includes, anextract unit configured to extract features of a first image based on anelectromagnetic wave in a first frequency band, an acquire unitconfigured to acquire motion information about the features of the firstimage, a classify unit configured to classify the features of the firstimage into a first group and a second group based on the motioninformation, and a remove unit configured to remove, from the firstimage, a signal corresponding to a feature of the first image thatbelongs to the first group.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a camera system according to a first exemplaryembodiment.

FIG. 2 is a flowchart illustrating an operation of the camera systemaccording to the first exemplary embodiment.

FIGS. 3A to 3D illustrate the camera system according to the firstexemplary embodiment.

FIGS. 4A to 4C illustrate the camera system according to the firstexemplary embodiment.

FIG. 5 is a schematic graph illustrating the camera system according tothe first exemplary embodiment.

FIG. 6 is a flowchart illustrating an operation of the camera systemaccording to the first exemplary embodiment.

FIGS. 7A to 7C illustrate the camera system according to the firstexemplary embodiment.

FIG. 8 is a flowchart illustrating an operation of a camera systemaccording to a second exemplary embodiment.

FIGS. 9A and 9B are schematic graphs illustrating an operation of thecamera system according to the second exemplary embodiment.

FIG. 10 illustrates a camera system according to a third exemplaryembodiment.

FIG. 11 is a flowchart illustrating an operation of the camera systemaccording to the third exemplary embodiment.

FIG. 12 illustrates a camera system according to a fourth exemplaryembodiment.

FIG. 13 is a block diagram illustrating a hardware configuration of thecamera system according to the fourth exemplary embodiment.

FIG. 14 is a block diagram illustrating a software configuration of thecamera system according to the fourth exemplary embodiment.

FIG. 15 is a flowchart illustrating an operation of the camera systemaccording to the fourth exemplary embodiment.

FIG. 16 is a flowchart illustrating a learning phase according to thefourth exemplary embodiment.

FIGS. 17A and 17B each illustrate a learning model used in the camerasystem according to the fourth exemplary embodiment.

FIGS. 18A and 18B each illustrate a camera system according to a fifthexemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the disclosure will be described in detailbelow with reference to the accompanying drawings. The followingexemplary embodiments are intended to embody the technical idea of thedisclosure, but do not limit the disclosure. While a plurality offeatures is described in the following exemplary embodiments, not allcombinations of features described in the exemplary embodiments areessential to the disclosure and the features may be arbitrarilycombined. The following exemplary embodiments illustrate an examplewhere an active camera system including a lighting unit is used as acamera system. However, a passive camera system including no lightingunit may also be used.

Some of the sizes and positional relationships of the membersillustrated in the drawings are exaggerated for clarity of description.In the following description, the same components are denoted by thesame reference numerals, and descriptions thereof may be omitted.

Inventors have found that in a case where a person wearing clothes ischecked based on an image captured by using an electromagnetic wave, theabsorption or reflection of the electromagnetic wave by the clothescauses noise, which deteriorates the image quality. The deterioration inthe image quality may also cause a deterioration in the accuracy ofdetecting a dangerous object concealed under the clothes. The presentexemplary embodiment is directed to providing a technique for reducingnoise in an image based on an electromagnetic wave.

A first exemplary embodiment illustrates an example where a camerasystem that uses an electromagnetic wave is used as an applicationexample of an image processing apparatus. The present exemplaryembodiment illustrates a case where a terahertz wave is used as theelectromagnetic wave. The wavelength of the terahertz wave is longerthan that of visible light and infrared light, and thus the terahertzwave is hardly affected by scattering of light from an object and hashigh transmissivity with respect to many materials. In contrast, thewavelength of the terahertz wave is shorter than that of a millimeterwave, and thus it is expected that the terahertz wave can be applied toan electromagnetic camera with a high resolution. It is also expectedthat an image inspection method using the terahertz wave, which has theabove-described features, can be used as a safe image inspection method,in place of X-rays. For example, it is expected that the imageinspection method using the terahertz wave can be applied to a securitycheck or monitoring technique in a public place. Typically, theterahertz wave is an electromagnetic wave having a signal in anyfrequency band, or including a single frequency, in a range from 0.1 THzto 30 THz. While the present exemplary embodiment illustrates an examplewhere the terahertz wave has a frequency of approximately 0.4 THz, thedisclosure is not limited to this example.

FIG. 1 illustrates the camera system according to the first exemplaryembodiment. The camera system includes an image processing apparatus101, a reception unit 102, and a radiation unit 103. The reception unit102 detects a terahertz wave from an object 106, and outputs a signalbased on the terahertz wave. The reception unit 102 can also be referredto as an image capturing unit or a camera. The radiation unit 103radiates a terahertz wave 104. The radiation unit 103 can also bereferred to as a lighting unit. As a detailed configuration of theradiation unit 103, a configuration discussed in Japanese PatentApplication Laid-Open No. 2014-200065 can be applied. The imageprocessing apparatus 101 processes the signal received from thereception unit 102. In this case, the image processing apparatus 101includes an image generation unit 108 and a processing unit 109.However, image generation processing may be performed by a singleprocessing unit, or at least a part of the processing may be performedon a cloud system.

The object 106 is a person. In this case, a coating material 105 may beclothing including fiber or leather. A concealed object 107 may be anyarticle, such as a dangerous object made of metal or ceramics. Theobject 106 is not limited to a person, but instead may be an article. Inthis case, the coating material 105 may be a wrapping, a packaging, orthe like made of, for sample, paper, cloth, or plastics. In the presentexemplary embodiment, the concealed object 107 is held by the object 106and is covered with the coating material 105. The concealed object 107is an object to be detected in the camera system.

A frequency range of the terahertz wave to be used will now bedescribed. In many cases, the coating material 105 is made of amaterial, such as clothing, which has high transmissivity with respectto electromagnetic waves of up to approximately 1 THz. In order toobtain an image resolution with which the shape of the concealed object107 can be identified, in one embodiment, an appropriate wavelength isused. The frequency of the terahertz wave with an appropriate wavelengthis approximately 0.3 THz. In one embodiment, the terahertz wave having afrequency range from approximately 0.3 THz to approximately 1 THz isused, accordingly. Thus, the terahertz wave used in the presentexemplary embodiment has a frequency of approximately 0.4 THz asdescribed above.

An operation of the camera system illustrated in FIG. 1 will bedescribed. The terahertz wave 104 is radiated from the radiation unit103. The object 106, the coating material 105, and the concealed object107 are irradiated with the terahertz wave 104. The most part of theterahertz wave 104 is transmitted through the coating material 105 andis reflected by the surface of the concealed object 107 and by thesurface of the object 106. The reception unit 102 receives the reflectedterahertz wave. The reception unit 102 outputs a signal based on thereflected terahertz wave. The image generation unit 108 generates imagedata from the signal based on the terahertz wave. The processing unit109 processes the image data output from the image generation unit 108.

In this case, a part of the radiated terahertz wave 104 is reflected bythe coating material 105. In other words, the reflected terahertz wave104 detected by the reception unit 102 includes information about thecoating material 105. The processing unit 109 performs processing forremoving the signal based on the terahertz wave reflected by the coatingmaterial 105 from the image data generated by the image generation unit108.

This processing will be described with reference to FIGS. 3A to 3C. FIG.3A is a schematic diagram illustrating a case where the object 106, thecoating material 105, and the concealed object 107 illustrated in FIG. 1are viewed from the front side. The coating material 105 is clothing.The clothing has a plurality of decorative objects 301 to 304. Thedecorative objects 301, 302, and 303 are, for example, buttons. Thedecorative object 304 is, for example, a pocket. As illustrated in FIG.3A, the concealed object 107 is covered with the coating material 105.FIGS. 3B and 3C schematically illustrate images of the object 106, thecoating material 105, and the concealed object 107 illustrated in FIG. 1when the images are captured from the front side. FIG. 3B is a schematicdiagram illustrating an image 305 when captured with visible light. FIG.3C is a schematic diagram illustrating an image 306 when captured with aterahertz wave. In FIGS. 3A to 3C, the illustration of the object 106,such as a person, is omitted for ease of explanation.

The image 305 illustrated in FIG. 3B indicates a coating material image315 and a plurality of decorative object images 311 to 314. The coatingmaterial image 315 and the plurality of decorative object images 311 to314 are visible light images of the coating material 105 and theplurality of decorative objects 301 to 304 illustrated in FIG. 3A. Incontrast, the visible light image corresponding to the concealed object107 illustrated in FIG. 3A is not illustrated in FIG. 3B. This isbecause the visible light is reflected and absorbed by the coatingmaterial 105, which makes it difficult to check the concealed object 107covered with the coating material 105.

The image 306 illustrated in FIG. 3C indicates a plurality of decorativeobject images 321 to 324 and a concealed object image 327. The pluralityof decorative object images 321 to 324 and the concealed object image327 are terahertz images of the plurality of decorative objects 301 to304 and the concealed object 107 illustrated in FIG. 3A. The terahertzimage corresponding to the coating material 105 illustrated in FIG. 3Ais not illustrated in FIG. 3C. This is because, while the terahertz waveis transmitted through the coating material 105, the decorative objects301 to 304 and the concealed object 107 can be reflecting objects forthe terahertz wave.

The coating material 105 is made of, for example, fiber. Theelectromagnetic wave in the terahertz wave band has high transmittancewith respect to fiber. In contrast, the electromagnetic wave in theterahertz wave band has a higher reflectance for the decorative objects301 to 303 and the decorative object 304, compared with the coatingmaterial 105. For example, if the decorative objects 301 to 303 arebuttons, the electromagnetic wave in the terahertz wave band has a highreflectance for the material of the buttons in many cases. Further,since the electromagnetic wave in the terahertz wave band locally has ahigh reflectance at, for example, wrinkles in the clothing, theelectromagnetic wave in the terahertz wave band has a high reflectancefor the decorative object 304, such as a pocket. The terahertz wave istransmitted through the coating material 105 and is reflected by thedecorative objects 301 to 304 and the concealed object 107, accordingly.Thus, the image 306 is an image obtained by superimposing an imagecorresponding to the terahertz wave reflected by the concealed object107 and an image corresponding to the terahertz wave reflected by eachof the decorative objects 301 to 304. In the present exemplaryembodiment, the concealed object 107 is an object to be detected. Inthis regard, inventors have found that although the concealed object 107can be detected by capturing an image using a terahertz wave, undesiredinformation, such as information about the plurality of decorativeobject images 321 to 324, is inevitably superimposed as noise. Theinventors have also found that the fact that the object 106 and theconcealed object 107 are moving at different speeds or cycles relativelyto the coating material 105 can be used to remove noise. Next, noisereduction processing to be performed by the image processing apparatus101 will be described.

Processing performed on the image acquired as described above will bedescribed with reference to FIG. 2. FIG. 2 is a flowchart illustratingprocessing performed in the image processing apparatus 101. Thisflowchart is started after the processing unit 109 receives image data.In step S201, features of a terahertz image are extracted. Examples ofthe features may include an edge, an angle, a singular point, and acontour. Other examples of the features may include a figure and an areaextracted by identifying edges or corners. As an example of thefeatures, an area obtained as a result of performing contour extractionusing a Laplacian filter is used. Any other known technique may also beused to extract the features. FIG. 3D is a schematic diagramillustrating an image 307 obtained by processing the image 306illustrated in FIG. 3C and performing contour extraction to identify thearea. In other words, the image 307 illustrated in FIG. 3D is an imageindicating the features. A feature 331 illustrated in FIG. 3D isgenerated from the decorative object image 321 illustrated in FIG. 3C. Afeature 332 illustrated in FIG. 3D is generated from the decorativeobject image 322 illustrated in FIG. 3C. A feature 333 illustrated inFIG. 3D is generated from the decorative object mage 323 illustrated inFIG. 3C. A feature 334 illustrated in FIG. 3D is generated from thedecorative object image 324 illustrated in FIG. 3C. A feature 337illustrated in FIG. 3D is generated from the concealed object image 327illustrated in FIG. 3C. A contour is extracted from each image and thearea of each image is extracted.

In step S202, motion information about the extracted features isacquired. In this case, the motion information can also be referred toas a feature amount. The motion information is, for example, a velocityvector of each feature. The velocity vector can be extracted from amovement vector and a frame rate of the reception unit 102. As a unitfor extracting the movement vector, a unit that extracts a movementamount from two or more images can be used. For example, the movementvector can be extracted from two temporally consecutive images. Theimages to be used for extraction are not limited to two consecutiveimages, but instead may be two images to be appropriately selected, ormay be moving images. As a method for extracting the movement vector,for example, a block matching method can be suitably used. Thisextraction processing will be described with reference to FIGS. 4A to4C.

FIGS. 4A to 4C illustrate the movement vector extraction processing.Each movement vector is extracted by tracking. FIG. 4A illustrates animage 411, which is one frame at a certain time. FIG. 4B illustrates animage 412, which is one frame at a time after the certain time. In thiscase, the image 411 illustrated in FIG. 4A corresponds to the image 307indicating one of the features illustrated in FIG. 3D. FIG. 4Cillustrates an image 413 indicating movement vectors extracted from theimage 411 and the image 412. In the image 413, the direction andmagnitude (speed) of each vector are indicated by arrows. A vector 401is a movement vector for the feature 331 in the image 307. A vector 402is a movement vector for the feature 332 in the image 307. A vector 403is a movement vector for the feature 333 in the image 307. A vector 404is a movement vector for the feature 334 in the image 307. A vector 407is a movement vector for the feature 337 in the image 307. The magnitudeof each vector indicates the movement speed of each feature. In FIG. 4C,for example, the magnitude of each of the vectors 401 to 404 is greaterthan 0, and the vectors 401 to 404 are each indicated by an arrow. Themagnitude of the vector 407 is 0. The vector 407 is represented by apoint. In the case of extracting movement vectors, the movement vectorsare extracted from images of two frames in the present exemplaryembodiment, but instead can be extracted from images of a plurality offrames. The movement speed can be extracted from a movement amount and aframe rate.

In step S203, the features are classified based on the extractionresult. In this case, the features are classified into Group 1 (Gr1) andGroup 2 (Gr2) that is different from Group 1. FIG. 5 is a schematicgraph illustrating the classification of features according to thepresent exemplary embodiment. Specifically, FIG. 5 illustrates themovement speed of each feature in a bar graph. The movement speed can bereplaced with a movement amount.

The features 331 to 334 are signals based on the decorative objects 301to 304 attached to the coating material 105. The movement speed of eachof the decorative objects 301 to 304 is more than or equal to athreshold, and thus these features are classified into Group 1 to becorrected. The feature 337 is a signal based on the concealed object107. The movement speed of the feature 337 is less than the threshold,and thus the feature 337 is not classified into Group 1 to be corrected,but is classified into Group 2.

FIG. 6 is a flowchart in which the classification step S203 illustratedin FIG. 2 is illustrated in more detail. Step S203 is a featureclassification step. Step S203 includes steps S601 to S603. In stepS203, motion information about each feature acquired in step S202 isused. Specifically, in step S601, it is determined whether the movementspeed of the feature is more than or equal to a threshold to classifythe feature. If the movement speed of the feature is more than equal tothe threshold (YES in step S601), the processing proceeds to step S602.In step S602, the feature is classified into Gr1. If the movement speedof the feature is less than the threshold (NO in step S601), theprocessing proceeds to step S603. In step S603, the feature isclassified into Gr2. The determination and classification processing canbe performed on all the extracted features, or can be performed on anyof the features. In the present exemplary embodiment, for example, awalking speed of approximately 4 km/h, or a time variation ofapproximately 1 mm/s in the breast of the person due to breathing is setas the threshold. In addition, the threshold can be set based on thenumber or distribution of features with a large movement amount. In thecase of FIG. 5, the features 331 to 334 are classified into Gr1, and thefeature 337 is classified into Gr2.

After that, as illustrated in FIGS. 2 and 6, in step S204, the signalbelonging to Gr1 is removed from the terahertz image. This removalprocessing is performed by, for example, replacing the signal belongingto Gr1 with a signal located at the same coordinates in another image.The other image to be used is not limited to a temporally consecutiveimage (continuous frame), and the temporal relationship between imagesis not particularly limited. If undesired objects, such as thedecorative objects 301 to 304, are superimposed with the concealedobject 107, an appropriate image is selected based on the result oftracking each feature. This processing will be described in detail withreference to FIGS. 7A to 7C.

FIG. 7A illustrates an image 701 that is temporally consecutive to theimage 306 illustrated in FIG. 3C. In other words, the temporallyconsecutive image 701 is an image of a continuous frame. FIG. 7Billustrates an image 702 that is not temporally consecutive to the image306 illustrated in FIG. 3C. In the case of removing the signal belongingto Gr1 from the image 306, the following processing is performed. Thesignal located at the same coordinates as the decorative object images321 to 324 is selected from one of the image 701 and the image 702. Theselected signal is replaced with the signal corresponding to each of thedecorative object images 321 to 324 in the image 306, thereby making itpossible to remove the signal belonging to Gr1. In this case, the image701 or the image 702 is also referred to as a frame to be replaced.

In the present exemplary embodiment, the image 702 is selected ratherthan the image 701 as the frame to be replaced. In a case of selectingthe image 701, some of the decorative object images 321 to 324 may beleft as indicated in an image 703 illustrated in FIG. 7C. The use of theimage, such as the image 702, in which the decorative objects 301 to 304move to a larger extent than the concealed object 107, facilitates theremoval of an undesired signal from the image 306. The image, such asthe image 702, can be extracted based on the above-described movementvectors.

Another removal method is, for example, a method of replacing the signalbelonging to Gr1 based on a surrounding signal of each feature. In oneembodiment, this method may be used in a case where the size of eachfeature to be removed is smaller than the number of pixels of the image.Specifically, the signal belonging to Gr1 is replaced with an averagevalue or median of pixel values of a background image. Alternatively,information about the shape of a portion that is not replaced can beinterpolated by performing interpolation processing based on the signalbelonging to Gr2, which is not a correction target. As a removal method,any combination of the above-described methods can be used.

Processing to be performed after step S204 includes the followingprocessing. That is, a shape or object can be identified using thefeature belonging to Gr2. Alternatively, a shape or object can also beidentified by extracting features again using an image based on thesignal belonging to Gr2.

The number of groups into which features are classified is not limitedto two, but instead may be three or more groups may be provided. In thiscase, a plurality of groups can be selected as groups to be removed froman image. Alternatively, the images from which groups have been removedmay be compared and the groups to be removed may be identified.

The processing described above makes it possible to acquire theterahertz image in which noise is reduced. The use of the terahertzimage in which noise is reduced makes it possible to improve theaccuracy of the camera system. Further, the processing according to thepresent exemplary embodiment facilitates calculation processing andfeature classification processing.

A second exemplary embodiment differs from the first exemplaryembodiment in that a movement frequency is used as motion informationinstead of a movement speed. In the following description, thedescription of processing similar to that in the first exemplaryembodiment is omitted.

FIG. 8 is a flowchart illustrating processing in the image processingapparatus 101 according to the second exemplary embodiment. Steps S201,S202, and S204 are similar to those in the first exemplary embodiment.In the present exemplary embodiment, after step S202, step S801 isprovided to perform Fourier Transform (FT) of a time variation in themovement amount of each feature. The movement frequency of each featurecan be obtained by performing FT of a time variation in the movementamount of each feature. That is, the movement frequency of each featureis used as motion information (feature amount) about each feature.

FIG. 9A is a schematic graph illustrating a time variation in themovement amount of two features with different movement cycles. Amovement amount 911 indicates, for example, the movement amount of thefeature 331. The movement amount 911 can be obtained by tracing themovement of the feature 331 or the decorative object image 321. Themovement tracing can be performed using a plurality of images capturedat different times. The movement amount 911 is, for example, a timevariation in the movement amount of a button on clothes. Not only themovement amount of a button, but also the movement amount of wrinkles ina pocket or clothes may be used. A movement amount 917 indicates, forexample, the movement amount of the feature 337. The movement amount 917can be obtained by tracing the movement of the feature 337 or theconcealed object image 327. The movement amount 917 is a time variationin the movement amount of the object 106, such as a person, or a timevariation in the movement amount of the concealed object 107 owned bythe object 106. In a case where the movement amount 917 indicates themovement amount of the object 106, if the concealed object 107 does notmove relatively to the object 106, for example, the object 106 is anarticle and the concealed object 107 is a fixed article. In a case wherethe movement amount 917 indicates the movement amount of the concealedobject 107, if the concealed object 107 is moving relatively to theobject 106, for example, the object 106 is a person and the concealedobject 107 is moved due to breathing of the person.

FIG. 9B is a schematic graph illustrating a histogram of the movementfrequency of each feature. A movement frequency 921 is extracted byperforming FT of the movement amount 911. A movement frequency 927 isextracted by performing FT of the movement amount 917. In this case, themovement frequency 921 is higher than the movement frequency 927. Thefeature 331 corresponding to the movement frequency 921 is attached tothe coating material 105 and fluctuates due to vibrations of theclothes, and thus has a high frequency.

In step S802, it is determined whether the extracted movement frequencyis more than or equal to a threshold. If the movement frequency of thefeature is more than or equal to the threshold (YES in step S802), theprocessing proceeds to step S602. In step S602, the feature isclassified into Gr1. If the movement frequency of the feature is lessthan the threshold (NO in step S802), the processing proceeds to stepS603. In step S603, the feature is classified into Gr2. In the presentexemplary embodiment, for example, an up-and-down movement cycle ofabout 1 Hz of the center of mass of the person during walking, or a timevariation of about 20 times/min in the breast of the person due tobreathing, is set as the threshold. Alternatively, the threshold may beset based on the number or distribution of features with a highfrequency. In the case of FIG. 9B, the feature 331 and the otherfeatures 332, 333, and 334 have a high frequency and are classified intoGr1, and the feature 337 is classified into Gr2.

After that, as illustrated in FIGS. 2 and 6, in step S204, the signalbelonging to Gr1 is removed from the terahertz image. The processingdescribed above makes it possible to acquire the terahertz image inwhich noise is reduced. The use of the terahertz image in which noise isreduced makes it possible to improve the accuracy of the camera system.The processing according to the present exemplary embodiment facilitatesthe feature classification processing regardless of the magnitude of themovement speed of an object.

In a third exemplary embodiment, a camera system that is different fromthe camera system according to the first exemplary embodiment is used.Specifically, in the present exemplary embodiment, a visible lightcamera is provided and feature classification processing is performedusing a visible light image. In the following description, thedescription of processing similar to that in the first exemplaryembodiment is omitted.

FIG. 10 illustrates the camera system according to the third exemplaryembodiment. The camera system according to the present exemplaryembodiment includes a camera 1001 for visible light, in addition to theconfiguration of the camera system according to the first exemplaryembodiment. The camera 1001 can be placed at a location adjacent to thereception unit 102. A visible image with an angle of view or orientationsimilar to that of the terahertz image can be acquired. The imagegeneration unit 108 generates an image from a signal based on theterahertz wave output from the reception unit 102, and generates animage from a signal based on the visible light output from the camera1001. Assume herein that the image generated from the signal based onthe terahertz wave is referred to as a terahertz image, and the imagegenerated from the signal based on the visible light is referred to as avisible image. For example, the terahertz wave includes a wavelength ina frequency band of 0.1 THz or more and 30 THz or less, and the visiblelight includes a wavelength of 300 nm or more and 750 nm or less. In thepresent exemplary embodiment, the terahertz wave and visible light areused, but instead two types of images based on electromagnetic waves indifferent frequency bands (i.e., wavelength bands) can be used. Thedifferent frequency bands may partially include an overlapping portion.

FIG. 11 is a flowchart illustrating processing performed in the imageprocessing apparatus 101 according to the present exemplary embodiment.Steps S201, S202, and S204 are similar to those in the first exemplaryembodiment. The processing according to the present exemplary embodimentdiffers from the processing according to the first exemplary embodimentillustrated in FIG. 2 in regard to step S1101, step S1102, and theclassification step S203.

In parallel with steps S201 and S202, the signal from the camera 1001 isprocessed. In step S1101, features of the visible image generated by theimage generation unit 108 are extracted. Examples of a featureextraction method include the known feature extraction method asdescribed in the first exemplary embodiment. Alternatively, methodsother than the method can also be used. Furthermore, a method differentfrom the method used in step S201 to extract features of the terahertzimage may be adopted. For example, since the visible light is reflectedby the coating material 105, a specific pattern on the surface of thecoating material 105 may be extracted as the features.

In step S1102, motion information about the features of the visibleimage is acquired. The motion information is a velocity vector of eachfeature. Examples of a method for acquiring the motion informationinclude the known motion information acquisition method as described inthe first exemplary embodiment. Alternatively, methods other than themethod can also be used. Further, a method different from the methodused in step S202 to acquire the motion information about the featuresof the terahertz image may be adopted. For example, in the case ofextracting a pattern as the features, the motion information can also beacquired by tracing the outer edge of the pattern.

Based on the results of steps S202 and S1102, the features areclassified in step S203. In step S203, the motion information about thefeatures of the terahertz image is compared with the motion informationabout the features of the visible image. Specifically, in step S1103, itis determined whether the movement speeds of the features of the twoimages are equal. If the movement speeds of the features are equal (YESin step S1103), the processing proceeds to step S602. In step S602, eachof the features is classified into Gr1. If the movement speeds of thefeatures are not equal (NO in step S1103), the processing proceeds tostep S603. In step S603, each of the features is classified into Gr2. Inthe classification processing, not only the method using the movementspeed of each feature, but also a method of referring to motioninformation about the features of the terahertz image corresponding tothe features selected from the visible image can be used.

The visible image has a higher resolution than that of the terahertzimage. Thus, in the visible image, the shape or a portion, such as abody or an arm, of the object 106 can be recognized by applying a knownobject recognition technique. In other words, processing to be performeddepending on a specific cycle or speed on the portion, such as settingof a threshold depending on the cycle of breathing, can be performed,for example, on an area recognized as the body of the person byidentifying the shape or target of the object 106 by using the visibleimage. Further, each processing area can be divided for each object thatmoves at an individual speed or cycle, and a threshold forclassification and classification conditions for each processing areacan be changed, to thereby improve the classification accuracy. Eachprocessing area can be determined with a predetermined size. It is alsopossible to perform the processing by reducing the number of pieces ofunnecessary information by selecting an area corresponding to an objectfrom each processing area. Thus, the processing load can be reduced andhigh-speed processing can be achieved as compared with the case ofprocessing the entire image. According to the present exemplaryembodiment, at least one of these beneficial effects can be achieved.

According to the present exemplary embodiment, the use of a visibleimage having a higher resolution and less noise than in a terahertzimage makes it possible to improve the feature extraction accuracy andthe accuracy of motion information about features.

A fourth exemplary embodiment illustrates an example where a machinelearning model is generated when the machine learning model is used inthe determination step. The use of the machine learning model in thedetermination step makes it possible to improve the determinationaccuracy. In the following description, descriptions of components inthe fourth exemplary embodiment that are similar to those in the otherexemplary embodiments are omitted.

FIG. 12 illustrates a basic configuration of a camera system accordingto the fourth exemplary embodiment. The configuration according to thefourth exemplary embodiment differs from the configuration according tothe first exemplary embodiment illustrated in FIG. 1 in that the imageprocessing apparatus 101 is electrically connected to each of a learningserver 1202 and a data collection server 1203. In the present exemplaryembodiment, the image processing apparatus 101 is connected to each ofthe learning server 1202 and the data collection server 1203 via anetwork 1201. Although not illustrated in FIG. 12, the camera system canbe additionally provided with the visible camera 1001 described abovewith reference to FIG. 10 and an environment parameter measurementdevice for measuring an environment parameter to be described below. Thedata collection server 1203 is a server that stores images generated bythe image processing apparatus 101 and learning data 1700 describedbelow. The learning server 1202 is a server that performs learning andestimation for classifying the features into groups (e.g., Gr1 and Gr2)as described above. Hereinafter, learning is also referred to as alearning phase and estimation is also referred to as an estimationphase. The learning phase and the estimation phase will be describedwith reference to FIGS. 13 to 17B.

FIG. 13 is a block diagram illustrating an example of a configuration ofan information processing apparatus 1300. The information processingapparatus 1300 may be, for example, the image processing apparatus 101,the learning server 1202, or the data collection server 1203. Theinformation processing apparatus 1300 includes a central processing unit(CPU) 1302, a read-only memory (ROM) 1303, and a random access memory(RAM) 1304. The information processing apparatus 1300 includes a harddisk drive (HDD) 1305, a graphics processing unit (GPU) 1309, and anInterface Control (IFC) 1306. The information processing apparatus 1300further includes an input unit 1307 and a display unit 1308. Thesecomponents are disposed on a system bus 1301. The present exemplaryembodiment illustrates an example where the image processing apparatus101, the learning server 1202, and the data collection server 1203 havethe configuration illustrated in FIG. 13. However, the configuration isnot limited to this example.

FIG. 14 is a block diagram illustrating a configuration of each of theimage processing apparatus 101, the learning server 1202, and the datacollection server 1203, and exchange of information in the learningphase and the estimation phase. FIG. 15 is a flowchart illustrating anestimation processing flow in the estimation phase. Steps S201 and S204illustrated in FIG. 15 are similar to those in the first exemplaryembodiment, and thus the descriptions thereof are omitted in the presentexemplary embodiment. While processing similar to that in the firstexemplary embodiment is carried out in the present exemplary embodiment,the present exemplary embodiment can also be applied to the method ofprocessing the visible image as illustrated in FIG. 11. FIG. 16 is aflowchart illustrating a learning processing flow in the learning phase.FIG. 17A is a conceptual diagram illustrating the learning phase, andFIG. 17B is a conceptual diagram illustrating the estimation phase.

The estimation phase will now be described. The outline of theestimation phase is mainly illustrated in FIGS. 15 and 17B. In theestimation phase, a feature 1711 extracted from the terahertz imageobtained by capturing an image of the object 106 covered with thecoating material 105 and the concealed object 107 is input to a trainedmodel 1702. A classification result 1712 obtained as a result ofclassifying the feature 1711 into one of two groups is output. Thisprocessing will be described in more detail. An algorithm for carryingout the processing flow illustrated in FIG. 15 is stored in the HDD 1305or the ROM 1303 of the image processing apparatus 101 and the learningserver 1202. The algorithm is loaded into the RAM 1304 and is executedby the CPU 1302 or the GPU 1309. In step S201, the image processingapparatus 101 extracts the feature from an image based on theelectromagnetic wave stored in the HDD 1305 or the ROM 1303. The featureis transmitted to the learning server 1202 via the IFC 1306 and thenetwork 1201. In step S1501, the learning server 1202 stores the featurein the HDD 1305 or the ROM 1303, and inputs the feature to the trainedmodel 1702. This feature corresponds to the extracted feature 1711illustrated in FIG. 17B. In step S1502, the learning server 1202executes the estimation using the trained model 1702. The learningserver 1202 outputs the classification result 1712 based on theestimation. The classification result 1712 is input to the imageprocessing apparatus 101 via the IFC 1306 and the network 1201. Theclassification result 1712 is stored in the HDD 1305 or the ROM 1303 ofthe image processing apparatus 101. In step S203, the CPU 1302 and theGPU 1309 in the image processing apparatus 101 classify the featurebased on the classification result 1712. In step S204, the imageprocessing apparatus 101 removes the signal belonging to Gr1 from theimage. Thus, the estimation phase is executed.

The learning phase will now be described. The outline of the learningphase is mainly illustrated in FIGS. 16 and 17A. In the learning phase,the learning data 1700 is input to the learning model 1701 to therebygenerate the trained model 1702. Specifically, the trained model 1702 isobtained by performing learning to obtain an algorithm for the learningmodel 1701 with high accuracy by machine learning. The learning data1700 includes a collected terahertz image. The learning data 1700further includes a visible image corresponding to the terahertz image,and an environment parameter. The learning data 1700 can be dataobtained by processing collected images or parameters, or can beextracted data. The learning data 1700 can also include training data.

A specific processing flow will be described. In step S1601, thelearning server 1202 requests the data collection server 1203 totransmit the learning data 1700. In step S1602, upon receiving therequest, the data collection server 1203 transmits the learning data1700 to the learning server 1202. In this case, the learning data 1700is stored in a data storage unit 1423 of the data collection server1203. When the request is received, the learning data 1700 istransmitted to the learning server 1202 via, for example, a datacollection/provision unit 1422, the IFC 1306, and the network 1201. Inthe learning server 1202, the learning data received by a learning datageneration unit 1433 is stored in the data storage unit 1434.

In step S1603, the learning data 1700 is input to the learning model1701, and in step S1604, learning is executed. In the learning phase,the trained model 1702 is generated. The learning model 1701 includes analgorithm for classifying features into two groups (Gr1 and Gr2).

The algorithm illustrated in FIG. 17B is stored in the HDD 1305 or theROM 1303 of the data collection server 1203 or the learning server 1202.The algorithm is loaded into the RAM 1304 and is executed by the CPU1302 or the GPU 1309.

Machine learning is executed by the CPU 1302 and the GPU 1309 of thelearning server 1202. The GPU 1309 can perform calculation processingeffectively by performing parallel processing on a larger number ofpieces of data. It may be therefore, the GPU 1309 is used when learning,such as deep learning, is performed a plurality of times using alearning model. In the present exemplary embodiment, the GPU 1309 isused in addition to the CPU 1302 in the processing to be executed by alearning unit 1532. Specifically, in the case of executing a learningprogram including a learning model, the CPU 1302 and the GPU 1309perform calculation processing in cooperation with each other. In theprocessing to be executed by the learning unit 1532, one of the CPU 1302and the GPU 1309 may perform calculation processing. The estimation unit1531 may also use the GPU 1309 similarly to the learning unit 1532.

As a specific algorithm for machine learning, methods such as a nearestneighbor algorithm, a Naive Bayes method, a decision tree, and a supportvector machine can be used. Further, deep learning that generates byitself a feature amount for learning and a coupling weightingcoefficient may be performed by using a neural network. For example, aConvolutional Neural Network (CNN) model may be used as a deep learningmodel. Any one of available algorithms as described above can be used,as need, and can be applied to the present exemplary embodiment.

Machine learning enables learning (batch learning) in which learning iscollectively performed using a preliminarily collected data set andfeatures are classified using the same parameters in subsequentprocessing, and also enables real-time learning (online learning) inwhich learning is performed based on captured moving images. It is alsopossible to provide an intermediate learning mode every time a certainamount of data is accumulated. As learning data, image capturing datathat is obtained in an environment suitable for learning data may beused, or data obtained in the same environment may be used, unlike in aninspection system to which this processing system is applied.

A learning data collection method will now be described. To collect amoving image based on an electromagnetic wave, the camera systemdescribed in the first exemplary embodiment and a known image capturingmethod can be used. An example where a terahertz wave is used as theelectromagnetic wave will now be described. Assume that, in the presentexemplary embodiment, a terahertz moving image obtained by capturing animage of a person wearing clothes is used as the terahertz image. Tocollect a visible moving image corresponding to the terahertz movingimage, the camera system described in the third exemplary embodiment anda known image capturing method can be used. In this case, the visiblemoving image is captured with an angle of view equal or similar to thatof the terahertz moving image, simultaneously with the terahertz movingimage. The environment parameter for the moving image includes climaticconditions and information about vibrations of the clothes, which is acoating material. Examples of the climatic conditions include thetemperature, humidity, and weather when the terahertz moving image iscaptured. Examples of the information about vibrations of the clothesinclude a body shape of a person, a wind speed, and the material ofclothes. To collect the environment parameter, a known data collectionmethod can be used. The environment parameter is not limited to thatdescribed above, but instead other various types of data may be used.Supervised data may be used as the learning data. The learning data iscollected by the data collection server 1203 via the network 1201.

The disclosure can also be implemented by processing in which a programfor implementing functions according to the above-described exemplaryembodiments is supplied to a system or an apparatus via a network or astorage medium, and a computer in the system or apparatus reads out andexecutes the program. The computer includes one or more processors orcircuits, and may include a plurality of separate computers, a pluralityof separate processors, or a circuit network so as to read out andexecute a computer-executable instruction. For example, the processorsor circuits may include a CPU, a micro processing unit (MPU), a GPU, andan application specific integrated circuit (ASIC). For example, theprocessors or circuits may include a field-programmable gate array(FPGA), a digital signal processor (DSP), a data flow processor (DFP),or a neural processing unit (NPU).

A fifth exemplary embodiment illustrates an example where a vibratoryunit that applies a mechanical vibration to an object during capturingof an image based on an electromagnetic wave is used to achieve animprovement in classification accuracy. The descriptions of parts in thefifth embodiment that are similar to those in the above-describedexemplary embodiments are omitted. FIGS. 18A and 18B each illustrate anexample of a camera system according to the present exemplaryembodiment.

The camera system illustrated in FIG. 18A includes a wind-powered device1801, in addition to the configuration described in the first exemplaryembodiment. The wind-powered device 1801 is composed of, for example, afan or an air conditioner, which enables the coating material 105 tovibrate with wind. With this configuration, the difference in themovement speed or movement frequency among the coating material 105, theobject 106, and the concealed object 107 can be increased, therebymaking it possible to improve the classification accuracy. The camerasystem illustrated in FIG. 18B includes a vibratory device 1802, inaddition to the configuration described in the first exemplaryembodiment. The vibratory device 1802 is provided below the object 106,but instead may be provided at any location as long as a vibration canbe transmitted to the object 106. With this configuration, the object106 is caused to vibrate, thereby enabling the vibratory device 1802 tocontrol the movement frequency or cycle of each of the object 106 andthe concealed object 107. Thus, an improvement in classificationaccuracy due to the difference in the movement frequency is achieved. Asdescribed above, the air flow of the wind-powered device 1801 and thefrequency of the vibratory device 1802 are reflected in the setting ofthe threshold for classification and the learning data 1700 duringmachine learning, thereby achieving an improvement in classificationaccuracy.

FIGS. 18A and 18B illustrate the wind-powered device 1801 and thevibratory device 1802, respectively, as examples of the device forapplying a vibration. Alternatively, a device that allows the object 106to actively vibrate may be placed. For example, a step is provided (astep device is provided) on a walkway for the object 106, thereby makingit possible to induce ascending and descending motions of the object106. The step device can be implemented in an inspection system withmore simplicity and lower cost than a system using the wind-powereddevice 1801 or a system using the vibratory device 1802.

According to the exemplary embodiments, the use of a visible imagehaving a higher resolution and less noise than in a terahertz imagemakes it possible to improve the feature extraction accuracy and theaccuracy of extracting motion information about features. The exemplaryembodiments can be arbitrarily changed or combined. The processing inthe image processing apparatus 101, the data collection server 1203, andthe learning server 1202 is not limited to the processing describedabove. The image processing apparatus 101 can receive a trained modelfrom the learning server 1202, and the image processing apparatus 101can execute the estimation phase. Further, the estimation phase and thelearning phase can also be performed on a cloud system. While themovement speed and the movement frequency are illustrated as examples ofthe motion information, any other information can also be used.

OTHER EMBODIMENTS

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

while the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2020-069430, filed Apr. 7, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An apparatus comprising: a first extract unitconfigured to extract features of a first image based on anelectromagnetic wave in a first frequency band; a first acquire unitconfigured to acquire motion information about the features of the firstimage; a classify unit configured to classify the features of the firstimage into a first group and a second group based on the motioninformation; and a remove unit configured to remove, from the firstimage, a signal corresponding to a feature of the first image thatbelongs to the first group.
 2. The apparatus according to claim 1,wherein the motion information about the features of the first image isa movement amount of each of the features of the first image, andwherein the classify unit classifies the features of the first imagedepending on whether the movement amount of each of the features of thefirst image is more than or equal to a threshold.
 3. The apparatusaccording to claim 1, wherein the motion information about the featuresof the first image is a movement frequency of each of the features ofthe first image, and wherein the classify unit classifies the featuresof the first image depending on whether the movement frequency of eachof the features of the first image is more than or equal to a threshold.4. The apparatus according to claim 2, wherein the motion informationabout the features of the first image is acquired from two or more ofthe first images captured at different times.
 5. The apparatus accordingto claim 1, further comprising a second acquire unit configured toacquire the first image, wherein the second acquire unit includes: aradiation unit configured to radiate the electromagnetic wave in thefirst frequency band; and a reception unit configured to receive theelectromagnetic wave in the first frequency band.
 6. The apparatusaccording to claim 1, further comprising: a second unit configured toextract features of a second image based on visible light in a secondfrequency band; and a third acquire unit configured to acquire motioninformation about the features of the second image.
 7. The apparatusaccording to claim 6, wherein the motion information about the featuresof the second image is a movement amount of each of the features of thesecond image, and wherein the classify unit compares the movement amountof each of the features of the first image with the movement amount ofeach of the features of the second image, and classifies the features ofthe first image depending on whether the movement amounts are equal. 8.The apparatus according to claim 6, wherein the motion information aboutthe features of the second image is a movement frequency of each of thefeatures of the second image, and wherein the classify unit compares themovement frequency of each of the features of the first image with themovement frequency of each of the features of the second image, andclassifies the features of the first image depending on whether themovement frequencies are equal.
 9. The apparatus according to claim 6,wherein the motion information about the features of the second image isobtained from two or more of the second images acquired at differenttimes.
 10. The apparatus according to claim 6, wherein the classify unitperforms processing on each of processing areas obtained by dividing thefirst image into a predetermined size, and wherein the processing areasare determined based on the features of the second image.
 11. Theapparatus according to claim 1, wherein the classify unit inputs thefirst image to a learning model to estimate a classification of each ofthe features from the first image.
 12. The apparatus according to claim11, wherein the learning model is generated by performing machinelearning based on an input of training data.
 13. The apparatus accordingto claim 1, wherein the remove unit replaces at least the signalbelonging to the first group with a signal located at the samecoordinates in another frame.
 14. The apparatus according to claim 1,wherein the remove unit replaces at least the signal belonging to thefirst group with a value based on a surrounding signal.
 15. Theapparatus according to claim 1, further comprising a vibratory unitconfigured to apply a mechanical vibration to an object during capturingof the first image.
 16. The apparatus according to claim 15, wherein thevibratory unit is at least one of a wind-powered device, a vibratorydevice, and a step device.
 17. The apparatus according to claim 1,wherein the electromagnetic wave in the first frequency band is aterahertz wave.
 18. A method comprising: extracting features of a firstimage based on an electromagnetic wave in a first frequency band;acquiring motion information about the features of the first image;classifying the features of the first image into a first group and asecond group based on the motion information; and removing, from thefirst image, a signal corresponding to a feature of the first image thatbelongs to the first group.
 19. A computer-readable storage mediumstoring a program for causing a computer to execute a method, the methodcomprising: extracting features of a first image based on anelectromagnetic wave in a first frequency band; acquiring motioninformation about the features of the first image; classifying thefeatures of the first image into a first group and a second group basedon the motion information; and removing, from the first image, a signalcorresponding to a feature of the first image that belongs to the firstgroup.